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Overview 


This report contains summaries of the four projects completed during the performance of this 
research. The four projects described are: 


1. Perceptual Augmentation Aiding for Situation Assessment 

2. Perceptual Augmentation Aiding for Dynamic Decision-Making and Control 

3. Action Advisory Aiding for Dynamic Decision-Making and Control 

4. Display Design to Support Time-Constrained Route Optimization 


Papers based on each of these projects are currently in preparation. The theoretical framework 
upon which the first three projects are based. Ecological Task Analysis, was also developed during 
the performance of this research, and is described in a previous report, and can also be found in 
Kir lik (in press). Requirements for psychological models to support design: Toward ecological 
task analysis, in Flach, Hancock, Caird. and Vicente (Eds.), The Ecology of Human- Machine 
Systems . Erlbaum. A project concerned with modeling strategies in human control of a dynamic 
system was also completed during the performance of this research, and is described in Kirhk 

(1993) Modeling strategic behavior in human-automation interaction: Why an aid can (and 

should) go unused. Human Factors, 35(2), 221-242. The following graduate students’s research 
was supported either all or in part, by this research: 


1. Ling Rothrock, MS, Ind & Sys Eng, 1991. Currently working toward PhD. 

2. Merrick Kossack, MS, Ind & Sys Eng, 1992. Currently with CTA. Inc. 

3. Laura Moody, PhD, Ind & Sys Eng, 1993. Currently Asst Prof at Mercer University. 

4. Wendy Markert, PhD, Ind & Sys Eng, Expected 1994. 

5. Wes Krosnick, MS, Ind & Sys Eng, Expected 1994. 



Perceptual Augmentation Aiding for Situation Assessment 

Alex Kirlik. Georgia Institute of Technology 


Problem definition 

In a variety of technologically complex environments, a major contributor to cognitive 
demands appears to be the presence of perceptually impoverished displays which provide only an 
opaque window to the task at hand. Such interfaces may require the operator to meet significant 
demands in order to develop and maintain an active understanding of the state of the task system 
(i.e., situation assessment) and also to ensure that action selection is consistent with both goal- 
centered and environmentally-centered constraints on productive behavior (i.e., dynamic decision 
making). The problem of "situation awareness" in modem vehicular control systems is a prime 
example of a limitiation on skill acquisition and performance apparently due to the lack of 
perceptual information at the interface capable of adequately specifying the environmental state and 
constraints on productive behavior. 

Given this observation, it is natural to wonder how much the acquisition of skill could be 
accelerated and final performance improved if the operator was provided with information displays 
which better specify both the state of the environment and the constraints to which action selection 
must be sensitive. We present research which attempts to address this question in the context of a 
dynamic laboratory decision and control task, the Extended Joint Surveillance and Target Attack 
Radar System (EJSTARS). First, task analysis was performed to identify a set of environmental 
and goal-centered constraints on productive action. Information capable of specifying these 
constraints was not found to be readily perceptible on the original EJSTARS display. Therefore, a 
second, perceptually augmented display was created which provides graphical information capable 
of specifying three environmental and one goal-centered constraint on productive behavior. An 
experiment was performed to evaluate the potential benefits of the perceptual augmentation 
approach in terms of enhancing both skill acquisition and peak performance. Since it is not always 
possible to enhance the actual displays in an operational environment, the experiment was designed 
to also evaluate training in terms of the ability of the augmented displays in the training system to 
support skilled decision and control performance even after the perceptual augmentation is 
removed. 

Domain description 

The research domain is a hypothetical information-gathering-and-attack system, referred to 
as the Extended Joint Surveillance and Target Attack Radar System (EJSTARS). It consists of a 
terrain-filled battlefield environment consisting of stationary friend targets that must be protected 
from unidentified enemy war-vehicles. These vehicles are located using an orbiting radar plane 
and may be identified or destroyed using unmanned aerial vehicles (UAVs), all ot which are 
controlled by a single operator from an interface inside a ground control station. The laboratory 
simulation is implemented on an Apple Macintosh Quadra 700 computer. A Radius PrecisionColor 
19" monitor is used at 1 152 x 870 resolution. All input occurs on a standard mouse input device. 
The computer language used for the simulation is Aldus' SuperCard 1.6. All 3-D graphics and 
animations are programmed using Specular International's Infini-D 2.0. 

Task analysis 

A task analysis based on the framework described in Kirlik (in press) was performed on 
the EJSTARS task. In this approach, the task is described in terms of the constraints upon 
productive action selection. The constraints may be both environmentally-centered (whether 
actions are consistent with the environmental structure), and goal-centered (whether actions are 



consistent with task goals or payoff structure ). The operator's task can then be considered to be 
selection of action which is mutually consistent with both these internal and external constraints. A 
display can therefore be evaluated in terms of the degree to which it makes perceptual information 
available capable of specifying these constraints. A process modeling approach which was 
successful in mimicking dynamic decision and control behavior based on such an approach is 
presented in Kirlik, Miller, and Jagacinski (1993), and a demonstration of the utility of this 
approach for display aiding in a dynamic decision and control task is presented in Kossack (1993). 

Constraint identification 

The first step in the task analysis is the identification of the environmental and goal-centered 
constraints to which behavior must be sensitive in order to be productive. The results of the task 
analysis suggested that three environmental, and one goal-centered, constraints were crucially 
important in the selection of action. The environmental constraints are: 

1. Constraints governing the locomotion of vehicles through the terrain. 

2. Constraints governing the distance enemy tracks can fire weapons. 

3. Constraints governing the penetrability of friendly armor by enemy weapons. 

The goal-centered constraint is: 

4. Constraints governing the relative value of enemy and friendly objects in terms 
of the task payoff structure 

Constraint 1 (locomotion) creates lawful regularities in the environment concerning 
possible paths of vehicles as a function of terrain (e.g., mountainous, level, ice covered). The 
operator must be sensitive with this constraint in order to correctly determine which enemy vehicles 
pose a threat to various friendly assets and which vehicles do not 

Constraint 2 (weapons range) creates lawful regularities in the environment concerning the 
ability of enemy vehicles to attack friendly assets as a function of distance between the enemy and 
friendly. As was the case with the locomotion constraint, the operator must be sensitive to this 
constraint in order to correctly determine which enemy vehicles pose a threat to various friendly 
assets and which do not. 

Constraint 3 (penetrability) creates lawful regularities in the environment concerning the 
ability of enemy vehicles to attact friendly assets as a joint function of the friendly armor thickness 
and the potency of enemy weapons. Once again, the operator must be sensitive to this constraint in 
threat/non-threat discrimination. 

Finally, even after all these constraints have been taken into account, it is quite possible that 
the environmental constraints underdetermine productive action, in that multiple threats may be 
encountered at the same time. In such cases, the operator must also select action in a manner 
sensitive to the payoff structure of the task, expressed above as Constraint 4. This final constraint 
allows the operator to select the most valuable action from the set of actions which are consistent 
with the three environmental constraints at any one time. 

Display evaluation 

The second step in the task analysis is the evaluation of the information display with respect 
to how well it supports the perceptual specification of the constraints identified above. The 
baseline EJSTARS display is shown in Figure 1. It was found that although the baseline display 
did provide information pertaining to all the task properties relevant to the four constraints, it did 
not provide integrated, relational information which could be directly used to select action in a 



manner consent with these constraints. An evaluation of the baseline display with respect to 
each of the four constraints is given below: 

Constraint 1 (Locomotion): 

The display provides terrain information in a graphical format (e.g., the actual EJSTARS 
display is a digitized photographic map of actual terrain). The display also provides information 
tSin/to eih veldcle me at a call up basis. Vehicle type information, in combinauon with 
S iXmauon, detennines where a vehicle can successfully locomote : e.g, certarn vehtcles can 
tocorriole over rocky or icy terrain while others cannot). Thus, while the display provides 
information that can be used to infer where a particular vehicle can and cannot locomote, the 
display does not provide the relational, integrated information that would allow locomotion 

possibilities to be perceived. 

Constraint 2 (Weapons Range): 

The display provides information on the location of the friendly assets and enemy vehicles 
through the use of icons superimposed upon the map. The display also provides inform atio 
pertaming to the weapons range of each vehicle type on a call up basis. The distance belw ^ 
K^^lsset and and enemy vehicle, in combinauon with the weapons range informauon tor the 
enemy^v^tcle^^rem^erwhether or not a a particular enemy vehicle poses a threat to a particular 
fSnfhv asset at a particular time. Thus, while the display provides the information that can be 
used to infer whether or not a particular enemy can attack a particular friendly asset at a P^cu ar 
USe, the display does not provide the relational, integrated informauon that would allow these 

attack possibilities to be perceived. 

Constraint 3 (Penetrability): 

The display provides information on the armor thickness of a friendly asset on a call up 
basis The display iLo provides information on the potency of a given enemy vehicle s weapons 
nrT^callup basis as well. These two items of information, in combinauon, determine whether or 
nm a a Darticular enemy vehicle poses a threat to a particular friendly asset at a particular time. 

T?,ut vffihe display provides information that can be used to infer whether or not a particular 
enonv^ weapons can penetrate the armor of a particular friendly asset, the display does not 
pS the Relational, integrated information that would allow these attack possibles to be 

perceived. 

Constraint 4 (Payoff) 

The riisnlav provides information on the number of personnel within every friendly vehicle 

wmmmmm 

in a perceptual fashion. 


Perceptual Augmentation 

The results of the task analysis suggested a number of deficiencies in the design of the 
baseline EJSTARS display, as described above In of die four task 

overly on ^ baseline 



EJSTARS display. The enhanced EJSTARS display is shown in Figure 2. Perceptual 
enhancements corresponding to each of the four task constraints are shown in the Figure. 

Locomotion Constraint: 

The locomotion overlay provides a color-coded map in the vicinity of an enemy vehicle 
which directly indicates whether or not that vehicle can locomote at each world location. This 
color-coded map is shown in blue and red at the upper right of the enhanced display around track 
"BMP-1." The blue areas indicate where this track may locomote and the red areas indicate where 
this track may not locomote. Similar locomotion maps are available for other tracks on a call-up 
basis. 

Range Constraint: 

The weapons range constraint is indicated by a small black circle centered at the location of 
each enemy vehicle. 

Penetrability Constraint: 

The penetrability constraint is indicated by coding penetrability as the thickness of the black 
walls around each friendly asset icon. Thin walls around a friendly asset indicate that the currently 
selected enemy track does not have weapons capable of penetrating the given asset's armor, while 
thick walls indicate that the given asset is vulnerable to the selected enemy track's weapons. 

Payoff Constraint: 

Payoff information for each friendly asset is indicated with small bar graphs on each asset. 


Evaluation 

An experiment was performed to evaluate the effectivess of this perceptual augmentation 
approach for enhancing skill acquisition and peak performance. The experiment also investigated 
the ability of perceptual augmentation to serve as a training tool by using transfer conditions which 
removed the augmentation from the display, to measure the degree to which performance might 
suffer without the presence of the augmented information. The experiment used 8 subjects per 
each display group (baseline vs. augmented). 

Results 

Figure 3 shows the mean overall score for both display groups for sessions 1-18. The 
augmented display accelerated the acquisition of skill as this group performed better than the 
baseline display group during the first three experimental sessions. During sessions 17 and 18. the 
number of tracks was doubled in order to assess the effects of perceptual augmentation during high 
workload conditions. Interestingly, while the performance of the augmented display group did not 
suffer during the two sessions of increased workload, the performance of the baseline display 
group decreased during these sessions, suggesting that perceptual augmentation promoted more 
robust behavior than did the baseline display. Finally, sessions 19 and 20 were performed by the 
augmented display group after the perceptual augmentation was removed. Removal of augmented 
information did not decrease the performance of this group. 


References 

Kirlik, A. (in press) Requirements for psychological models to support design: Towards 
ecological task analysis. In Flach. Hancock. Caird. and Vicente (Eds.), The Ecology of Human- 
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perceptual Augmentation Aiding for Dynamic Decision-Making and Control 

Alex Kirlik, Georgia Institute of Technology 

Problem definition 

In a variety of technologically ' p^-o'S 

demands appears to be the p ^ J s P uc h interfaces may require the operator to 

operator only an opaque ^ order t0 develop and maintain an active understanding ot the 

meet significant cogmuve demtm _ ssmen t) and also to ensure that action selection is 
state of the task system (i.e., pnvironmentally-centered constraints on productive 

consistent with both ^ g) Many cases of "human error" in the control of 

behavior (i.e., dynamic decisio^m^n^. stemming from the problem that readily available 

technologically Comdex situa . , ^ operator about the environmental constraints upon 

interface informauon does not w ^f/°^^i° s So ns , operators must keep knowledge of 
acceptable solutions to decision • constraints upon acceptable flight routes, vehicle 
environmental constraints in memory ( •S- because information capable of specifying 

performance characteristics. dec.sion , risks that fail to meet 

SUCh ^«SmaSy S«<l constraints governing acceptable system operatton can otten 

acquisirion of skill couid be 

acceler^d^^^^m^tTo°t^uc^if^^o^mtor^^proW^^wkh^irdomtauon^hspl^'S which 
better specify the environment^ consramb i in [he conu:xt of a dynamic laboratory 

present research which attempts to analysis was performed to identify a set ot 

decision and control task, St ^,^ t dlts upon pro ducdve action in this task. Informauon 
environmental and g°^ cen ^ constraints w P as not found to be readily perceptible on ^ baseline 
capable of specifying all of these c , Uv au2me nted display was created which 

StaiCruiser display. Therefore , ,a • P ^ lbe constraints upon each of the eleven 

provides graphical information behavioral repertoire. An experiment 

primitive actions that comprise the S^mtse rope . augmentation approach m terms 

was performed to evaluate it is not always possible to 
of enhancing both skill acquisitio ^ imnment the experiment was designed to also 
enhance the displays in M0 ^^f^ a5 ^ ity of [he augmented display to support skilled 
SrXXSrS perceptual augmentatton is removed. 


Domain description 

The research domain is a Ztr o?sS sysrems to 

sr>ace context The task environment consists ot z&ixy sub f ecL Each solar system 
which activity is confined, and a ^^ounts of scientific data and mineral resources, 

contains a number ot planets contain 8 fi nd these data and resources, to load them upon 

and it is the subject s goal to ( 5 x P l ^„ t ^ b !t ; y l0 eam points. Constraints on time and fuel 
the StaiCruiser, and to take item to a 1 “ e J of gathering data and resources provide the 

availability, and constraints The laboratory task is implemented on a 

main source of dynamic decision P g rec ; s ionColor 19" monitor. All input occurs with a 

for rhe usk is Lrghrspeed C. 



Task analysis 

A task analysis based on the ‘ofdwconstramts upon 

the StaiCruiser task. In this app^ch^e^ £ environmenta ii y -centered (whether 
productive action selecuon. The constraints may o 0 al-centered (whether acuons are 

KS?« consistent with d» can then be considered to be 

consistent with task goals or P a yo consistent with both these external and internal 

conlro1 " based on such 1 lheory 15 

presented in Kirlik, Miller, and Jagacinski (1993). 


Constraints on action 

percepttS^^nentation S^lo^tttr^ovi^'hddrTnadon^a^^^^coiwtmhits on^ese°actions 
are shown below. 


Stnr r mi* pr Action 
1. Deploy collection tool 


nkplnv Enh ancements 

1 a) Place tools on colored field 

lb) Add background colors to tools 

lc) X-out tools which cannot be deployed 

l d) Dynamic coloring of depoyment string 


2. Retrieve collection tool 


3. Deploy probe 


4. Retrieve probe 

5. Place StaiCruiser in orbit 


2 2 «« 

2c) Realign StaiCruiser capacity guages 

3 a) X-out tools which cannot be deployed 
3b) Dynamic coloring of deployment suing 

4a) Dynamic coloring of retrieval string 
5a) Add dynamic 


6. Remove StaiCruiser from orbit 

7. Dock StaiCruiser at Star Base 

8. Release StaiCruiser from StarBase 

9. Change display view 

10. Choose StarCruiser's destination 


No enhancements 

7a) Dynamic coloring to indicate docking speed 
No enhancements 
No enhancements 

10a) Color coding of suns to indicate commodiues 
10b) Blue X for empty solar systems 


, 1. Determine StaiCruiser mode Ua, Dynanuc adoring of SiatCtuiser oudine 

Each of the display enhancements listed For example, placing 

SSSiSSSSSSSSS SKSES 



legally deployed provides information about constraints upon acceptable (tool, planet) 
assignments. Dynamic coloring of the deployment and retrieval strings (Id, 2a, 3b, 4a) indicate to 
the operator when a distance constraint (between the mouse cursor and a specified target) has been 
met to enable tool deployment and/or retrieval. Each of the other enhancements provides similar 
constraint-based information. A depiction of both the baseline(top) and perceptually augmented 
(bottom) StarCruiser displays is shown in Figure 1. 

Evaluation 

An experiment was performed using both baseline and augmented subject groups, with 
eight subjects per group. Each subject received initial training and then performed StarCruiser for 
14 sessions. For sessions 15 and 16, subjects were also required to perform a concurrent mental 
arithmetic task in order to diagnose whether the augmented display lowered the cognitive demands 
experienced by subjects using the enhanced display. For sessions 17 and 18, subject groups 
switched displays, in order to diagnose whether subjects trained using the augmented skills would 
maintain their level of performance after removal of the augmented information. 

Results 

Figure 2 shows performance curves for both display groups for sessions 3- 14 (sessions 1 
and 2 were training sessions and were excluded from the analysis). Over these sessions, the main 
effect of display was significant with the augmented group performing better than the baseline 
display group. However, for sessions 13 and 14 only, the augmented display group and the 
baseline group did not differ in performance, suggesting that the primary benefit of the augmented 
displays in this experiment was the facilitation of skill acquisition. 

A set of 12 additional performance measures were constructed to individually evaluate the 
effect of each of the 12 perceptual augmentations of the display. Here, it was found that 8 of the 
12 display enhancements significantly improved performance of the augmented display group 
subjects, with the other 4 enhancements showing no effect on performance. The enhancements 
which proved to improve performance were: 

1. Dynamic coloring of deployment/retrieval string 

2. Realignment of StarCruiser capacity guages 

3. X indicating orbit radius 

4. Dynamic coloring of StarCruiser indicating orbit speed 

5. Dynamic coloring of StarCruiser indicating docking speed 

6. Coloring suns to indicate commodity levels 

7. Blue X indicating empty solar systems 

8. Dynamic coloring of StarCruiser to indicate mode 

The results of the concurrent mental arithmetic sessions are shown in Figure 3. Although the 
graph indicates that the performance of the baseline group decreased 27 % whereas the performance 
of the augmented display group did not change, this interaction did not reach significance. The 
results of the display-transfer sessions are shown in Figure 4. which indicates that neither group 
was affected by using the alternate display. Most interestingly, subjects using the augmented 
display, which was shown to facilitate skill acquisition in this task, did not suffer when the 
augmented information was removed. 

References . _ , 

Kirlik, A. (in press) Requirements for psychological models to support design: Iowards 
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Kirlik, A., Miller, R.A.. and Jagacinski, R.J. ( 1993) Supervisory control in a dynamic, uncertain 
environment: A process model of human-environment interaction. IEEE Transactions on 
Systems. Ma n, and Cybernetics, Vol 23., No. 4. 
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Action Advisory Aiding for Dynamic Decision-Making and Control 

Alex Kirlik, Georgia Institute of Technology 


Problem definition 

One approach for aiding human decision-making in complex human-machine systems is to 
provide the operator with a computer-based decision-aid which gives the operator advice by 
providing a dynamic, prioritized list of control actions which serve as recommendations. 
Presumably, the motivation behind this approach is that humans sometimes err in making decisions 
that could be better made by an algorithm implemented on a computer. However, it is recognized 
that in some cases the human will have access to information unavailable to the algorithm, so the 
human's task is not totally automated; rather, the human and computer are expected to work in 
tandem to hopefully produce performance superior to either the human or computer working alone. 

Indeed, the design of die AERA family of aiding systems currendy being considered for the 
US air traffic control system is based upon exactly this advisory aiding philosophy. However, a 
number of questions should be raised concerning the potentially effectiveness of this approach. 
Will the operator take the advice of the aid? Will human decision-making effectiveness degrade if 
the aid becomes temporarily unavailable due to technical failure? What will happen if the aid fails 
and gives faulty advice? In order to address questions such as these, an experiment was conducted 
using a laboratory dynamic decision and control task which compared the decision-making and 
control performance of a group of unaided subjects with a group of subjects for whom an action- 
advisory decision aid was made available. In addition to investigating the effect of the aid on skill 
acquisition and performance, the experiment also examined the effects of the aid becoming 
unavailable (both early and late in practice) and also the effect of the aid failing (providing 
erroneous advice) late in practice. 

Domain description 

The research domain is a hypothetical exploration and resource acquisition task set in a 
space context The task environment consists of a galaxy containing a number of solar systems to 
which activity is confined, and a spaceship under control of the subjecL Each solar system 
contains a number of planets containing various amounts of scientific data and mineral resources, 
and it is the subject's goal to explore the galaxy to find these data and resources, to load them upon 
the StaxCruiser, and to take them to a starbase to earn points. Constraints on time and fuel 
availability, and constraints governing acceptable ways of gathering data and resources provide the 
main sources of dynamic decision-making complexity. The laboratory task is implemented on a 
Macintosh Dei computer with a Radius PrecisionColor 19" monitor. All input occurs with a 
standard mouse device. The computer language used for the task is Lightspeed C. 

Action Advisory Aiding System 

A window at the bottom of the StarCruiser display was created to communicate the 
advisory aid's dynamic list of recommended actions to the subject. The aid itself is based upon a 
model of human dynamic decision-making which has proved capable of high levels ot performance 
in such tasks, and also of successfully mimicking human behavior (Kirlik, Miller, and Jagacinski. 
1993). In the model, each of the primitive actions available in the StarCruiser task is evaluated at 
every time step with respect to two criteria: a) the degree to which the action is consistent with the 
current environmental structure; and b) the degree to which the action is consistent with current 
task goals. Simple linear and boolean functions measuring the degree to which these two classes 
of constraints are met are used to determine a scalar value indicating the desirability of taking each 
action at each point in time. In the aiding system, the top four actions with respect to this measure 



are displayed to the subject in a prioritized list The internal effectiveness of the aid was validated 
by creating a version of the aid which was capable of performing the StarCruiser task without 
human intervention. This version of the aid achieved scores comparable to those achieved by local 
expert subjects. 

Experiment 

An experiment was performed using both baseline (unaided) and advisory aid subject 
oroups, with eight subjects per group. Each subject performed StaiCruiser for 28 sessions. The 
first four sessions consisted of active training by the experimenter and were therefore excluded 
from analysis. In sessions 9 and 10. 19 and 20, and 27 and 28, for the advisory aid group, the aid 
was made unavailable by removing the advice from the display. In sessions 23 and 24 for the 
advisory group, the aid was intentionally designed to give faulty advice; in this case, the worst four 
possible actions rather than the best four possible actions were presented, as measured by the 
method described above. 

Results 

Figures 1 and 2 show the mean performance of the two groups for the 24 sessions 
analyzed. The advice aid did have a positive effect on skill acquisition, as the aided group 
performed better than the unaided group over sessions 5-8 (the first four sessions after active 
training). There was some suggestion in the data that removal of the aid for sessions 9 and 10 
decreased performance of the aided group although this effect was not statistically significant. 
Removal of the aid during sessions 19 and 20 (labeled 9tr and lOtr on the graph) and during 
sessions 27 and 28 (labeled 17tr and 18tr on the graph) had no effect on the performance of the 
aided subjects. Thus, while it is still an open question whether aid unavailability decreased 
performance very early in practice, aid unavailability clearly had no negative effect on performance 
after subjects had a good amount of experience performing the task. The optimistic interpretation 
of these results is that temporary unavailability of an advisory aid is of little concern. On the other 
hand, these results can also be interpreted to suggest that operators made very little use of the aid, 
especially late in practice, a conclusion that raises serious doubts about the ability of such an aiding 
system to compensate for faulty human decision and control performance. 

Indeed, the findings for sessions 23 and 24 (labeled I3tr and 14tr on the graph) are also 
consistent with the interpretation that subjects made little explicit use of the aid. Recall that in these 
sessions, the aid was designed to intenuonally present faulty advice. However, clearly the subjects 
did not follow the faulty advice of the aid, as the performance of the aided group did not decrease 
during these two sessions. We also have anecdotal data concerning these two sessions that are 
quite relevant to achieving an interpretation of these data. Subjects were not informed in any way 
prior to these two faulty-aid sessions that these sessions were any different than those experienced 
previously. Only one of the eight subjects in the aided group made a spontaneous comment to the 
experimenter indicating that he realized that the aid was presenting faulty advice. Upon asking all 
eight subjects in the aided group after the completion of these sessions if they had noticed anything 
unusual, only one additional subject indicated that he realized that the aid was presenting faulty 
advice. Thus, six of the eight subjects apparently never even noticed that the aid had failed, 
lending support to the conclusion that these aided subject made very little, if any, explicit use of the 
information provided by the aid while performing at asymptotic levels of performance. Such 
findings raise serious doubts about the ability of such aids to compensate for poor human decision 
and control performance in tasks such as these. 

References 
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Display Design to Support Time-Constrained Route Optimization 

Alex Kirlik, Georgia Institute of Technology 




Problem definition 

A variety of human-machine systems require a human operator to create efficient solutions 
to constrained routing problems. Examples would be a logistics planner r^nsible for creating 
delivery truck routes to minimize travel costs, an emergency services dispatcher responsible for 
creatine vehicle assignments and routes to minimize travel time, and an air traffic controller 
responsible for scheduling cost efficient routes for aircraft. Each of these problems differs m terms 
of the cost function the operator is attempting to minimize and also with respect to the constrain 
which SrSe the spa£ of feasible solutions. However these usksall share the same general 
structure, which is that of a constrained vehicle routing problem. Such problems can someumes b 
solved via operations research optimization techniques, however, most problems m operauonal 
environments are either too complex or else not sufficiently closed to enable a computer-based 
algorthmic solution to be found. In such systems, a technique called interacuve optimization is 
freauentlv employed to allow a human to interact with an optimizatton system to presumably obtain 
problem solutions superior to those that could be achieved by either the human or computer acung 
alone Design issues for interactive optimization systems can be expected to attract increasing 
attention as increases in computer power begin to allow some aspects of even ’ very complex 
operational routing problems (e.g., terminal area flight route scheduling) to be treated 
algorithmically, thus creating a situation where more and more human operators will be asked to 
interact with computers to solve problems that were once the provence of human expert alone 

Although there is a large and rapidly growing number of papers on the design of interactive 
optimization systems in the operations research and logistics literature, rarely if ever do these 
papers ever refer to the vast literature on issues such as human problem-solving, human-computer 
interaction, and other issues which can be expected to be quite relevant to the effective design ot 
these interactive systems. In our review of the interactive optimizauon literature, it became 
apparent that one reason for this disconnect is that the operations research community ts simply 
unaware of the existing body of knowledge on psychological issues m design, ^*"dic at ed . 
authors presenting ad hoc designs and solely intuitive justifications for these designs, as if a y 
of teowEe relevant to these decisions did not exist. However, it also became dear that the 
human problem-solving, human factors, and human-computer interacuon literatures also had a role 
tov\?y ui°his regSd: & types of problem solving tasks addressed by these literatures are vastly 
oversimplified as compared to the types of problem solving tasks for which interactive systems 
must be designed to support human operators in operational environments The purpose ot the 
SntShwas to begin to bridge the gap between the tasks studied by both basic psychology 
research and human factors research, and the tasks faced by operators performing constrained 
vehicle routing problem-solving in operational environments. 

Domain description 

Vehicle routing problems (VRP's) in operational environments differ in at least two 
imnortant respects from the types of problems which have given nse to both theories ot hum 
problem-solving (mostly well structured puzzles and games) and design guidelines to support 
nroblem-solving behavior (frameworks based on these theories). First, in a vehicle routing 
Zbte” mteatnnot describe problem-solving as the creation of a set ot aettons m order to move 
from an initial slate to a goal slate within a state space deleted at a single J iphys.ca ) leve ot 
nhstraction Even the simplest of VRP's. the Traveling Salesman Problem (TSP). requires tne 
problem-solver to generate a solution (the shortest path connecting all destinations) which is not a 
sinele state in the physically defined problem space (such as can be done in games such as the 
Tower of Hanoi where the goal state is one of the many physically possible states of the puzz ). 



Rather in the TSP the goal is the shortest path, which is an abstract measure of goodness upon 
routs which are themllves the objects capable of being represented in die physical proWem 
Sat Many different possible routes may result in the same measure of goodness . and ^the goal I ts 
to find any one of the many possible routes which minimizes travel distance This difference in the 
formal representations of traditionally studied problem solving tasks and tasks such as the VRP 
creates many difficulties when trying to apply traditional problem-solving theories to VRP s. 

A second criucal difference between the VRP and the problem-solving tasks typically 
studied by researchers concerns the intensive perceptual nature of human interacuon with the VR . 
Existing problem-solving theories characterize mental acuvity as the construction of an internal 
problem representation, and all problem solving is done via the internal manipulation (e.g., means- 
ends analysis) using this represenuon. In the VRP, on the other hand, it is clear that perceptual 
activity operating directly upon information in the external problem representation (or disp ay), 
provides^ crucial contribution to problem-solving activity. Various patterns and features of die 
disDlav are recognized and perceptually structured, and problem-solving acuvity involving the 
manipulation of an internal representation seems to be called upon only when this perceptual mode 
of problem solving fails due to conflicts needing to be resolved by considering the problem in a 
more abstract fashion. For example, even a 100-destination TSP could be solved in a pure y 
perceptual manner if the set of 100 destinations happened to be distributed in a clear, circular 
fashion, which would allow the problem solver to quickly sketch in the circle as the optimal route. 

Given the desire to preserve these two features of VRP’s that make them d ^ ffe ^ r °^ v ^p 
tvoes of problems that have been traditionally studied, we chose the time-windowed VRP ( 

TW) S as the problem domain for this research. The VRP-TW is more complicated than the simple 
TSP in that each destination may have associated with it a ume window that specifies theearhest 
and latest times at which the destination can be visited along the route. We chose the VRP-TW 
since it contains this additional time constraint characteristic of problems in many operational 
environments, and also because this time constraint creates a problem of such complexity that 
efficient algorithms for solving this problem are not available, thus necessitating an interac v 
optimzation design approach. Although this problem is much more complex than those problems 
which have motivated most cognitive theories of problem solving (and therefore is not readi i y 
amenable to analysis in terms of these theories), it is nevertheless sull much simpler than most real- 
world VRP tasks. However, our goal in this research was to take initial steps at building a n g 
between the basic and applied worlds of human problem solving, and we are not aw-are of any 
Studies oTproblem solving which have addressed a task of the complexity studied here. 

Design approach 

Our first studies in this research concerned humans solving the simple unconstrained TSP 
in order to motivate an understanding of how problem solving was performed in this percepwatiy 
intensive task Our initial results indicated that problem solving appeared to be performed using 
TteZetLcZXwen perceptual and higher-level cognitive activities. Since it appmddiat 
the very efficient perceptual activities appeared to make a crucial contribution to performance and 
£t Se activities were called upon to resolve conflict that had to be treated in a 

we naturally decided to pursue a design approach which tostered a 

Sww . solving is rendered impossible 

because in the traditional representation the time window information is presented numerically by 
nwEaZ indicating the earliest and latest visiting times, next to each desunauon on the 
graphical map. In seeking to make these time-constraints perceptually measurable, we chose to use 
a three-dimensional representation of the problem, in which the third, vertical dimensio 
rpnrp^nted time Thedesign which resulted from this approach is shown in Figure 1. The top 
SK shows m the SdidorS representation of Ihe problem, here with one posable so luuon. m 
which time constraints are represented numerically. The bottom display shows the three 
dimensional representation of the same problem, where the route is shown traveling upw 
time as it moves away from the point of origin. In this display, ume windows aie y 



displaying vertical bars at each point which is time constrained. Red regions of the bar indicate 
dmes which are either too early or too late for visiting the associated point, and green regions ot the 
bar indicate acceptable visiting times. Note from the two displays the vast difference in effort 
shnDlv to make a judgment of route feasibility. In the top display the operator must read the time 
at which the route visits each time constrained point, and then read the early and late times 
assodated with this point, and ensure that the visiting time falls between the early and late times. 

In the lower display, the operator only has to ensure that the line showing the routedoes not pass 
through a bar at a point at which the bar is red. rather than green. Experiments conhrmed that 
feasibility judgments were indeed made more more quickly and accurately with this three- 

dimensional displ^.^H ^ p ^ 0 f shifting the information processing burden to 

perception is greatly reduced when the problem becomes so complex that clutter and masking make 
[h e required perceptual judgments difficult to perform. Although, as shown on the lower display, 
subjects were able to rotate and tilt the problem representation, when 35 time windows were 
present it was extremely difficult even to make feasibility judgments with the three dimensional 
disolav We thus sought a design solution which exploited the benefits of graphical representauon 
of time window constraints but did not result in unacceptable levels of clutter ^and masking. 

Our solution to this problem is shown in Figure 3 which shows the VRP Solver system. 

In the colored, circular portion of this display the top-down view of the layout of destinations is 
shown as on the traditional VRP-TW display. However, the VRP solver system also includes (at 
the bottom) a graphical display that once again shows ume in a verucal dimension. In this portion 
of the display, the current route (the current solution at some point in the problem solving Process) 
is shown as a line from the origin point in the lower left comer, to the last city at the right .of the ) - 
axis at increasing heights on the Y-axis indicating the times at which each ot the points along the 
route are reached. Vertical bars are used to indicate feasible times at which the points can be visited 
(black areas indicate times which are too early or too late). The color of each of the bars is identical 
to the color of the region of the point's location in the upper map representauon. This consistent 
coding allows problem-solver to better associate the information contained within the two 
rep^ntaS As indicated by the menu of options at the left side of the screen (and can be better 
seen in the line drawing shown in Figure 4), the problem solver can take acuons such as moving a 
point to a different place within the route, group and ungroup points so they can be moved together 
as a unit, add and delete points from the current route, etc. In addition, a local improvement option 
can be engaged (via the Improve item in the menu) which allows the problem solver to have the 
computer make improvements to the route based upon an optimization algorithm. 

Experimentation 

Figure 5 summarizes the differences between the traditional (Table) display and the VRP 
Solver (Graph) display in terms of the types of inferences that must be made in order to solve the 
VRP-TW As indicated by the figure, the Graph display was designed to make many °f the 
necessary inferences performable in a perceptual fashion. In order to evaluate the potenual benefit 
of thi^design approach, an experiment was performed using 10-subject groups, performing on 
either the bSe Table display or the VRP Solver graphical display. Subjects were required to 
create routes as short in distance as possible while meeung as many time constraints as possible. 

Results 

Figure 6 (top) shows that subjects using the graphical display produced routes with a 
significantly lower percent of points visited late. More striking was the tmding that subjects using 
the graphical display produced^outes with a total lateness approximately one-fifth that of the routes 
Droducedbv the subjects using the table display (total lateness was measured as the sum ot the 
Liount of time late at each point over the enure route). These findings provide some support to 
design approach for complex routing problems which attempts to shift information process g 

burdens to the perceptual system. 
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l ime Windows on (a) 2D and (b) 3D Displays 
(Five Time Windows) 
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VRP Solver: An Interactive System for Solving the 
VRP/TW Visually. 




VRP Solver: An Interactive System for Solving the VRPmV 
(Line Drawing). 




fvai IIATF ROUTE UbltKMUNt REVERSAL IDENTIFY CUSTOMERS 

EV ALU A IE KUU I b LOCATION 


Inferences Required to Solve the VRP/TW & Information Provided 
in the Two Display Conditions. 


INFERENCE 


GRAPH DISPLAY 


TABLE DISPLAY 


What is late 

What is most late 

What is early 

What is very early 

Most constrained 
(narrow) 

What takes 
precedence 

Is there a pattern 


Route line passes above 

Distance from route line to 
customer 

Route line jumps up to customer 
Distance of jump up to customer 
Length of customer line 

Comparisons among customers 

Relative size and location of 
customer lines 


Table entry in red 

Difference between arrival time 
and latest time 
Two entries in yellow 

Difference between arrival times 
in first and second entry 
Differences between listed 
latest and earliest times 

Listed latest, earliest, and 
arrival times of TWs 


Is reversal potentially 

beneficial | Patterns of narrowness & lateness 

Location of customers 

i 

Will there be a 
negative impact 


How much earlier/later 

Effect on other 
customers 


Color of entries, listed latest 
earliest, and arrival times. 


TWs met that were not 

TWs no longer met 

More or less difficult 
to meet TWs. 


General slope of route line, 
distance from customer line, 
color of length of customer 
lines, and location of route 
line with respect to other 
fWs. 


Listed latest, earliest, and 
arrival times. 

Number and location in table 
of red/yellow entries 


Location of route line with 
respect to TWs, length of 
customer lines, distance 
between customer line 
and route line. 


Number and location in table 
of red/yellow entries. 
Listed latest, earliest, and 
arrival times. 







Mean Dependent Measure by Display 




